Binaural audio generation via multi-task learning

We present a learning-based approach for generating binaural audio from mono using multi-task learning. Our formulation leverages additional information two related tasks: the generation task and flipped classification task. learning model extracts spatialization features visual input, predicts left right channels, judges whether channels are flipped. First, we extract ResNet video frames. Next, perform separate subnetworks based on features. method optimizes overall loss weighted sum of losses tasks. train evaluate our FAIR-Play dataset YouTube-ASMR dataset. quantitative qualitative evaluations to demonstrate benefits over prior techniques.